HDInsight Tool for IntelliJ is GA!

  • Article

We are excited to announce that the HDInsight Tool for IntelliJ is now GA. The HDInsight Tool for IntelliJ serves the open source community and will be of interests mainly to HDInsight Spark developers. It has seamless integration with IntelliJ and enables users to enjoy the native support for the Scala and Java Spark application development experience.

For users who are developing Spark applications on HDInsight, it could be very hard to get started building a first application, and the development cycle of Spark application from IDE could be long and tedious. With this tool, users can get started with HDInsight Spark in a few minutes. Experienced Spark developers can iterate their development cycle faster and easier.

Tool Overview

This tools extends IntelliJ to support Spark job life cycle from create, author, debug and submit job to Azure cluster and view results.  This IntelliJ HDInsight tool integrates well with Azure to allow user navigate HDInsight Spark clusters and view associated Azure storage account. To further boost productivity, the IntelliJ HDInsight tool also offers the capability to view Spark job history, display detailed job logs, and the job output to boost developer productivity. A few usability improvements have been implemented upon user preview feedback, which includes auto locate artifact, add intelligence to remember assembly location, caches spark logs, etc.

The HDInsight Tool for IntelliJ offers the following capabilities:

  • Create Spark project with built-in templates, sample code and intelligence to auto locate artifact and assembly.
  • Develop your Spark applications with native authoring support (g. IntelliSense, auto format, error checking, etc. ) from IntelliJ (mainly Scala and Java).
  • Submit the Spark applications to HDInsight clusters.
  • Integrate with Yarn and Spark to view job history, job logs and job results.
  • Allow user to stop a running Spark application before its completion.
  • Test and validate the Spark application locally.
  • View all the Spark clusters associated with your azure subscriptions.
  • Navigate the associated storage resources of your HDInsight Spark cluster.
  • Cache the Spark logs into disk in case the Spark logs are too big.

 

Create, Author, Submit and Stop a Spark Project

The HDInsight Tool for IntelliJ support Spark Project full life cycle management from creation to submission and the ability to view job results with seamless integration with Azure.

Create a new Spark Project

saveas1 saveas2

View Spark Cluster and Navigate the associated storage account

navigate1

Project Submission

submit1

Support Stopping Spark Applications in IntelliJ UI

UI1

After submitting Spark jobs to HDInsight, the tool allows user to stop a running Spark application in IntelliJ before it’s completion for whatever reasons.

Installation

You can get the latest bits by going to IntelliJ repository, and search “HDInsight”. IntelliJ will also prompt you for latest update if you have already installed the plugin.

installation1

 

For more information, check out:

 

If you have questions, feedback, comments, or bug reports, send email to hdivstool at microsoft dot com.